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PREFACE 

\ 
J 

This report was prepared as part of Rand's DoD Training and Man- 
power Management Program, sponsored by the Human Resources Research 
Office of the Defense Advanced Research* Projects Agency (ARPA) . With 
manpower Issues assuming an ever greater Importance in defense planning 
and budgeting. It Is the purpose of this research program to develop 
broad strategies and specific solutions for dealing with present and 
future military manpower problems. This Includes the development of 
new research methodologies for examining broad classes of manpower 
problems, as well as specific problem-oriented research. 

The report describes the use of a computer program based on "ad- 
missible probability measurement," a testing procedure that permits a 
user to express his (or her) degree of certainty as to the correctness 
of alternative answers to multiple-choice questions. The program is 
implemented on PLATO IV, an interactive computer-based education system 
available at many military sites. ^J'lAfSi ^KHY l**^ Vr-X? f f* 

Two previous Rand reports prepared under the DbD Training and Man- 
power Management Program provide additional Information on admissible 
probability testing. Rationale of Computer- Administered Admissible 
Probability Measurement^ R-1371-ARPA, by E. H. Shuford, Jr., and T. A. 
Brown, presents the theory behind admissible probability testing. An 
experimental computer implementation of admissible probability testing 
techniques at The Rand Corporation is described in A Prototype Computer 
Program for Interactive Computer- Administered Admissible Probability 
Measurement^ R-1258-ARPA, by W. L. Sibley. Also for the purpose of 
testing the theory,- a version available on a portable, stand-alone 
terminal was developed based on the prototype implementation. 

This report is a guide for instructors and students interested 
in using computer-aided admissible probability measurement (CAAPM) on 
PLATO IV. The instructional text in Sees. Ill and IV is also avail- 
able online: the tutorial for students is found in lesson randS on 
PLATO; the author guide in lesson randlO. 
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SUMMARY 



This report provides instructions on how to create and take 
computer-aided admissible probability measurement (CAAPM) tests using 
programs available on PLATO IV. Admissible probability measurement, 
as discussed in Sec. I, is a testing procedure that permits a user to 
express a degree of uncertainty as to the correctness of alternative 
answers. 

Section II describes PLATO IV, an interactive computer-based edu- 
cation system on which CAAPM resides. Section III presents a tutorial 
for students on how to take CAAPM tests, demonstrating how the standard 
test- taking approach may force the student to guess an answer, whereas 
CAAPM lets the student express his (or her) uncertainty. Section IV 
tells instructors how to create CAAPM tests easily and quickly on the 
PLATO IV system, and Sec. V discusses recommended extensions to CAAPM. 
For example, the use of slides, touch panels, and random-access audio 
devices could enhance the testing capabilities of the system. Student 
data collection and analysis should also be added. The external valid- 
ity graph Is illustrated and explained in App. A, and standard tests 
available in CAAPM format on Plato IV are included in App. B. 

5f 

Sections III and IV are also available online in lessons rand3 
and randlO, respectively. 
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I> INTRODUCTION 

When a student submits an essay answer to a question he Is graded 
as If he believed 100 percent in that answer. On a multiple-choice 
type of test he is asked to select one right answer. In either case 
the student may guess, and thus his response will not reflect his true 
state of knowledge on the subject. Admissible probability measurement 
(APM) is a testing procedure that permits a user to express his degree 
of certainty or uncertainty as to the correctness of his answer. 

The multiple choice type of test is used in the current implemen- 
tation of APM. Three possible answers are displayed at the corners of 
a triangle. The student moves a cursor around this space toward or 
away from the possible answers. The spatial relationship of the cursor 
position to each corner reflects the user's beliefs as to the correct- 
ness of each answer. This approach permits use of APM without ex- 
tensive explanation on subjective probability. 

Effective use of APM dictates its implementation on an inter- 
active computer system. It was first made available at Rand on the 
Video Graphics System [1] . This version of APM, however, could only 
be demonstrated at Rand. We next developed APM on an IMLAC, a stand- 
alone portable terminal using cassettes for input and output. This 
enabled us to demonstrate APM both here and at other sites. The final 
version, described in this report, resides on the PLATO IV system, an 
interactive computer-based education system available at many military 
sites [2]. In addition to being widely available for demonstration, 
APM may now be used by instructors and students in actual courses. 

It is the purpose of this report to provide instructions on how 
to create and take admissible probability tests using programs avail- 
able on PLATO IV. The report includes; 

o A description of the computer system, PLATO IV, on which 
the CAAPM programs reside and an explanation of how to use 
these programs. 

o A tutorial for students on how to take CAAPM tests. 

o A guide for authors on how to create CAAPM tests. 
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A discussion of recommended extenaions to the current CAAPM 
design. 

An illustration of the external validity graph. 
Sample tests using CAAPM available on PLATO IV. 
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II > COMPUTER SYSTEM DESCRIPTION 



PLATO IV SYSTEM 

The PLATO system is an interactive computer-based education system 
developed over the last decade by the Computer-Based Education Research 
Laboratory (CERL) at the University of Illinois. In its fourth version 
the system handles more then 700 terminals at over 100 locations through- 
out the United States, Canada, and Europe. There are presently more 
than 2000 lessons available in 65 fields of study [3]. Students have 
logged over 25,000 contact hours of instruction at all school levels on 
PLATO IV [2]. This version of PLATO is specifically designed to lower 
the terminal/computer costs to 25-30 cents per student contact hour. 
For additional information on the history and design of the system, see 
Ref. 4. 

The major hardware components and the communications options for 
linking the components of the PLATO IV system are identified in Fig. 1 
[21. 

PLATO TERMINAL AND KEYBOARD 

Whether the user is ^ student or an author of CAAPM, he communi- 
cates to PLATO via the keyboard (Fig. 2). Students use the six arrow 

0 0 0 0 CD 0 

keys to move the cursor.' These keys may be pressed directly without 
the use of -SHIFT-. The -NEXT- and -DATA- keys are pressed to con- 
tinue through the lesson. Authors use the standard keyboard characters 

v' 

to enter questions and answers; the -NEXT- key to continue; the SHIFT- 
STOP to try a test; and -BACK- to back out of a lesson when editing. 

The user's communication with PLATO is displayed at the terminal. 
The parts of the terminal that are used with CAAPM are identified and 
explained in Fig. 3 [2]. 
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Fig. 1 — Components of Plato IV system 
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Screen (pIcBma panel) : Displays text and graphics. 

Terminal clear switch: Resets the circuitry in the terminal and 

erases the screen. Use when screen has 
been cluttered with garbage and/ or the 
error reset switch has not corrected the 
communication error. 

Error reset switch: Resets the circuitry that indicates a non- 
correctable communication error hos occurred. 

Error light: Turns on (red) when communication errors occur. 

Power switch: Turns tenninal on and off. 

Fig. 3— PLATO IV terminal 
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GETTING TO CAAPM 

PLATO may be used only if it recognizes the user's name, course, 
and password. To enter this information into the system and to request 
lesson space for new CAAPM tests, authors should contact CERL (217) 
333-2374, Students should contact authors for permission to use PLATO 
and for the name of CAAPM test lessons* Once signed onto the system, 
a lesson name must be entered. Sections III and IV direct the reader 
to existing example CAAPM lessons. 
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STUDENT GUIDE 

How to take tests using CAAPM will be discussed In this section. 
An online version of this tutorial Is available In lesson rand3 on 
PLATO. 

CAAPM represents a new concept In test-taking. You will have an 
opportunity to use your knowledge In a novel way to answer questions. 
Consider the following analogy question: 
Angle Is to degree as (ANGLE : DEGREE :: ) 
1* area: square Inch 

2. tlmermlnute 

3. milk: quart 

Standard test-taking procedures ask you to respond with one choice. If 
you are certain that the first answer, for example. Is the correct an- 
swer, then your response accurately reflects your knowledge. However, 
If you think the correct answer might be the second or third choice, or 
If you have no Idea which Is the correct answer, the standard approach 
forces you to guess and your response no longer reflects your knowledge 
of the subject. CAAPM Is a test-taking procedure that lets you express 
your uncertainties as well as your certainties. 
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For each question you will see a triangle with the possible answers 
displayed near each comer (see Fig. 4). At the beginning of each ques- 
tion a point will be displayed In the center of the triangle* By mov- 
ing this point within the triangle or along the lines toward or away 
from a possible answer you will Indicate your belief as to the correct- 
ness of that answer. 



Question: ANGLE: DEGREE: 



area: square inch 




time: minutes 



milk: quart 



Fig. 4 



17 



-10- 



To move the point, use the six arrow keys on the keyboard (see Fig. 
2), The direction In which each arrow will move the point Is demon- 
strated In Fig. 5. Because responses must be given within the triangle, 
you will not be able to move the point outside the triangle. 



Question: ANGLE : DEGREE : : 

area: square inch 




p Fig. 5 
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As you move the point toward the triangle your potential score Is 
displayed at each corner. The scores displayed at the beginning of 
each question when the point is located in the center of the triangle 
are 0,0»0 as shown in Fig. 6. 
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The maximum potential gain you can make is 23, while the maximum 
potential loss is 76. These scores are achieved by moving the point 
to a specific corner. For example, in Fig. 7 the cursor has been moved 
to the "arearsquare inch" comer. The 23 means you will gain 23 points 
if the answer "area:square inch" is the correct one. But if you have 
selected "arearsquare inch" and it is not the correct answer, you will 
lose 76 points. 

Scores do not reveal the correct answer. The number at a corner 
only indicates your potential gain or loss. 



Question: ANGLE : DEGREE : : 

area: square inch 
23 



-76 -76 
time: minutes milk: quart 

Fig. 7 ' 
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As you can see, you can lose many points if you choose one answer 
and it is wrong. Moving the point all the way to one comer is appro- 
priate only if you are certain of the correct answer. Thus if you are 
not certain which is the correct answer you may wish to reduce your 
potential loss (as well as gain) by moving the point away from a spe- 
cific comer. Compare Figs. 8 and 9. Notice that by moving away from 
the corner just a little you reduce your potential loss by 50 percent 
(76 to 37) while reducing your potential gain by only 10- percent (23 
to 21). 



area : square inch 
23 




-76 -76 



time: minutes milk:quart 
Fig. 8 



area: square inch 
21 




-37 -37 
time: minutes milk: quart 

Fig. 9 
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If you knew, however, that mllkiquart was not the correct answer, 
but you were unsure as to which of the other two were correct, you would 
move the point to the line between time : minutes and areaisquare inch 
as shown in Fig. 10. 




If you have no idea what the correct answer might be, you may 
choose to receive no points for the question by positioning the point 
in the center of the triangle (as in Fig. 5). 

You should not guess if you do not know for sure what the answer 
is. The scoring system is designed to heavily penalize a wrong answer 
and to encourage you to express your degree of certainty or uncertainty. 

To continue to the next question, you press the -NEXT- key. This 
instruction will appear at the bottom of each display. If the instruc- 
tion "press -DATA- for score" also appears, you may check your score 
on individual questions by pressing -DATA-. This is an option set by 
your instructor. 



See Ref . 5 for an explanation of the theory behind the scoring 
sys tems • 
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Your response is not recorded until you press -NEXT- or -DATA-; 
therefore you may adjust the point until you are satisfied with the set 
of scores* 

If you press -DATA- and your instructor has set this option, a dis- 
play like Fig. 11 will appear. Your score on the current question is 
flashed next to the correct answer. If, for example, you pressed 
-DATA- after moving the cursor to the point shown in Fig. 9, the score 
"21" would flash as shown in Fig. 11 because "area: square inch" is the 
correct answer. If, however, "time imlnutes" were the correct answer, 
the "-37" would flash. Your cumulative score, the sum of scores achieved 
on all questions up to and including the current one, is plotted on 
the graph in rislation to the maximum and minimum cumulative score that 
could be attained on the test as each question is answered. 

X Question: angle: degree: : 



oreo: square inch 




-37 -37 



time : m inutes milk: quart 

Fig. 11 

If you press -DATA- and your instructor has not set this option, 
the display will not change. To continue, press -NEXT-. 
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At the completion of the test you will see an assessment of your 
test performance. An example of a test performance assessment is given 
in Fig. 12. In the following paragraphs the info-nnation presented in 
a test performance assessment is interpreted. For an explanation of 
the derivation of the performance diagram and scores, see App. A. 




Information 



You tend to overvalue your information. 

You can improve your score by 37 points 
by more realistic use of your knowledge. 

You can improve your score by 224 points 
by more study. 

You can improve your score by 261 overall. 
Fig. 12 

The graph that appears compares how much you thought you knew 
(perceived) with an estimate of how much you really knew (revised) on 
the test. The difference between these two measures indicates your 
bias. Most people tend to be biased in the use of their information; 
that is, they tend to overestimate or underestimate the validity of 
their facts and reasons. You should try to learn to correctly assess 
the value of your facts and reasons, and thus eliminate all bias from 
your responses. 
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If you are successful at this, the line of the information graph 
will tend to be horizontal for any test you take (Fig. 13). If, how- 
ever » you overvalue your information the line will slope downward (Fig. 
14); if you undervalue your information, the line will slope upward 
(Fig. 15). 



P 
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V 
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d 

Infonmafion 
Fig. 13 — Correcfly valued 

P 
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r 
c 
e 
i 

v 
e 

Informafion 
Fig. 15 — Undervalued 
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I 




Information 
Fig • 14 — Overval ued 
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* The height of the line on the information graph indicates your 
knot^ledge of the subject matter. In Fig. 16, the student knew very 
little about the test subject and would gain considerably by more study, 
In Fig, 17, the student knew the subject matter well* In both cases 
he correctly valued his information, • 




Information 




Information 



Fig. 16 — Subject matter difficult 



Fig, 17 — Subject matter easy 



One reason for striving toward removing bias from your response 
is so that you can accurately communicate degrees of certainty or un- 
certainty not only to yourself but also to others. Another reason is 
that you lose points by being biased. In addition to a graphic display 
of your performance, CAAPM provides a numeric evaluation of this loss 
by: 

1. Estimating a revised test score that you would have made if 
your responses had been unbiased. 

2. Subtracting your test score from this revised test score. 

3. Reporting the result as '*You can improve your score by 

points by more realistic use of your information." 
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When you can Improve your score In this manner by less than 10 
points, CAAPM displays under the graph, "You tend to correctly value 
your information." When this potential improvement is 10 or more 
points, CAAPH displays, ''You tend to overvalue your information" or 
"You tend to undervalue your information," as the case may be. 

You can and probably will lose some points as a result of lack of 
conqplete information about the subject matter of the test or about the 
intent of the writer of the test questions. CAAPM evaluates this loss 
by: 



1. Computing the best possible score for the test (in this case, 
23 points per question). 

2. Subtracting your revised test score from this maximum score. 

3. Reporting the result as, "You can improve your score by 

points by gaining more information about the subject matter." 



Your overall improvement score, equal to the sum of the other two 

improvement scores, is reported by, "You can improve your score by 

points overall." You can do best by striving to maximize your total 
test score. 

To gain experience with CAAPM try taking one of the following prac- 
tice tests: 



Lesson 

rand4: International Relations 

rand6 : Statistics 

rand7: Art 

rand 8: Music 

rand9 : Li terature 
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IV- AUTHOR GUIDE 

How to create CAAPM tests will be discussed in this section; in- 
structions are also available on PLATO in lesson randlO. 

Your new test will be written in the TUTOR language used by the 
PLATO system [6]. Although you will not have to write extensively in 
the language, you should note the format of TUTOR statements. 

^±te A Test on International Relations 

The first part of statement (write) is called the command. The re- 
mainder (A Test on International Relations) is called the tagV Vou 
may tab over to enter a tag by pressing the key_ 
side of the keyboard. ^.^--^'^^^ 

To begin, type all statements as they appear inside the box in 
Fig. 18 into your lesson. The universal delimiter is created by 

pressing the micro key and then the comma key (see Fig. 2). There are 
seven items you add to this list of statements to create a new test: 

1. The number of questions in the test. 

2. A score flag. 

3. The title of the test. 

4. The test questions. 

5. The correct answers to the questions. 

6. One set of wrong answers. 

7. A second set of wrong answers. 

These items are entered using the edit command INSERT in the posi- 
tions indicated by the numbers outside the box in Fig. 18. For instruc- 
tions on the use of INSERT and other edit commands, press -HELP- while 
in your lesson. An example of a completed test with these seven items 
included is given in Fig. 19. 



..CR 
TAB 



on the left 
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(1) 
(2) 



(3) > 



(4) > 



(5) > 



(6) 



(7) 



unit 


begin 


goto 


one 


use 


randly caapt 


use 


caap,tl ' ' 


use 


caapf:2 


- use 


caapt 3 


use 


caapt 4 


* 




unit 


title 


at 












unit 


ques 


wrl tec 








unit 


corr 


wrltec 


num^^^^^^ 


* 




unit 


wrgl 


wrltec 


num 






unit 


wrg2 


wrltec 




* 





Fig. 18 — The skeleton test lesson 
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unit 
define 

goto 

use 

use 

use 

use 

use 

unit 

write 

at 

write 

unit 
wrltec 



* 

unit 
wrltec 



unit 
wrltec 



unit 
wrltec 



begin 
nq»5 

score»l -] ' 

one 

randlycaapt 

caaptl 

caapt2 

caapt3 

caaptA 

title 

A Test on International Relations 
3025 

press -NEXT- 
ques 

num^^^^^^ 

Selling goods for export at prices lower than 

those charged domestic buyers Is an example of^^ 

The communist theorists expound today's world 

movements as a triumph for the revolutionary concept of^^ 

Identify the most Important policy advisory body that 

works directly with the Secretary of Defense:^^ 

The decision as to which countries the U.S. shall 

recognize diplomatically lies with the^^ 

What do we call the strategy of buying goods to prevent 

a rival nation from getting them 



corr 



num^ 



dumping ^►Lenln^^Armed Forces 

Policy Councll^rfresldent^^preemptive buying 

wrgl 

subsidy ^►Marx^^Natlonal Security 
Councll^i^Senate ^►loading 



wrg2 
num^i 



embargo^i^Stalln^^De/ense Research and 
Engineering Group^^Secretary of State^^exribargo 



Fig. 19 — A completed test lesson 
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To specify the number of questions in your test you insert the in' 
struct ion 



conaniind tag 



define 



nq- 



entering after the equal sign the appropriate nuinber. In Fig, 19, the 
number of questions was set equal to 5. In this implementation of 
CAAPM, the maximum number of questions per test lesson is 9, 

As the author, you may choose to reveal or not to reveal to the 
student the correct answer to each question. You do this by defining 
the variable score, equal to 1 for yes (display score) and 0 for no 
(do not display) : 

define score » ^ 

In Fig. 18, score»l, and therefore the correct answer to a question 
will be displayed when the student presses the -DATA- key. 

To specify the title of the test you must insert a write command 
after the "unit title" instruction and then type in the title. The 
title may be up to 20 lines in length, each line containing a maximum 
of SO characters. The write conmand is entered only on the first line 
of the title. For each subsequent line, press -NEXT- and tab directly 
over to the tag field and continue the title. In Fig. 19, the title 
"A Test on International Relations" was entered. 

The next four items are concerned with the questions and answers 
for your test. The first question in the question list is associated 
with: the first correct answer in the correct answer unit "corr"; the 
first wrong answer in the first wrong answer unit "wrgl"; the first 
wrong in the second wrong answer unit "wrg2". The fifth, then, in 
each list is associated with the fifth in the other three lists. If 
you write out your test as shown in Fig. 20, you will have created the 
appropriate tags for each of these units. 
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All questions are inserted in the unit "ques" sequentially. The 
questions must be separated by the universal delimiter "^►" as shown 
In Fig. 19. Each question may be up to six lines in length, 60 charac- 
ters per line. 

Answers are inserted in units "corr", "wrgl", and "wrg2". Enter 
answers In an order to match the questions. As with the questions, 
each answer must be separated from the next by the universal delimiter 

Each answer may be up to two lines in length, 20 characters 
per line. For attractive display of answers taking two lines, press 
-NEXT- at an appropriate midpoint in your answer and continue on the 
next line. In the first example below, a break occurs between Policy 
and Council; in the second, between Forces and Policy. The latter 
alternative is preferred for display. 



dumping Lenin Armed Forces Policy 
Council President preemptive buying 

dumping 4^ Lenin ^> Armed Forces 

Policy Council President -^^ preemptive buying 



Display 

Armed Forces Policy 
Council 

Anted Forces 
Policy Council 



After all questions and answers have been inserted, you are ready 
to try your new test (press -SHIFT- and -STOP- simultaneously). Each 
time you try your test, note that CAAPM randomizes the order of ques- 
tions and the location of answers in order to minimize fallacious asso- 
ciations on the part of test repeaters. 



V, RECOMMENDED EXTENSIONS TO CAAPM 



INPUT/OUTPUT DEVICES 

CAAPM uses the keyboard for input and the plasma panel for output. 
The addition o£ several other input and output devices available on the 
PLATO system could enhance the use of CAAPM. 

In the current implementation, the student moves the cursor on the 
screen by pressing the arrow keys on the keyboard. A touch panel is 
available that enables a point to be indicated on the screen by simply 
touching, with pencil or finger, that point. The touch panel works on 
the photoelectric eye principle. It divides the screen area into 
16 X 16 « 256 half-inch squares [2]. For CAAPM use, however, only ap- 
proximately 50 squares would be active— that is, those squares that 
fall within the boundaries of the equilateral triangle. This resolu- 
tion, similar to the current arrow-key version which recognizes approxi- 
mately 50 locations within the triangle, is sufficiently accurate. 
Touch response is especially effective with young children. 

In Sec. IV we described how an author could enter a text-type 
question into CAAPM. But what if the author needs to display a pic- 
ture (perhaps complex, perhaps even color!) with, or instead of, a text 
question? For example, a pathology instructor may want to qxilz stu- 
dents on visual recognition of diseases by showing them photographs 
of various growths. Available on PLATO is a slide selector which can 
select any one of 256 full-color slides rapidly and project the image 
onto the terminal screen. Overlaying or beneath this picture on the 
screen, then, would be the CAAPM-generated interactive response tri- 
angle. 

Effective testing in some areas — for example, in phonics and con- 
versational French — requires the use of audio questioning. PLATO hafi 
available a random-access audio device that can select in a half second 
or less one of up to 4000 messages stored on a 15-inch flat disc. The 
total message time on one disc is about 23 minutes [2]. 

Use of these devices with CAAPM would require modifications to the 
existing program. 
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INSTRUCTION CAPABILITY 

CAAPM is currently designed to administer t^sts only. Addition 
of selective feedback of remedial Information based on user^s assess- 
ment probabilities would allow CAAPM to be used for Instruction, as 
well as testing* This would Involve expansion of the author mode to 
support remedial Information Input as well as changes to the program 
to permit selective branching. 

TEST EFFECTIVENESS ANALYSIS 

Creating effective tests — that Is, tests that measure what one 
wants them to measure — demands systematic analysis of questions and 
answers. This analysis might include, for example, application of the 
Ease Index (a measure of the ease of a question) and the Differentia- 
tion Index (a measure of the effectiveness of a question In differen- 
tiating among students) to the test [7]. However, the influence of 
guessing on these measurements has limited their usefulness. Since the 
CAAPM approach eliminates guessing, adding these types of measurements 
to CAAPM should enable authors to create better tests and, as a result, 
to collect more accurate and Informative data on student performance. 

STUDENT DATA COLLECTION AND ANALYSIS 

PLATO IV provides a means for collecting specific typ^s of data 
on students, for example, the number of wrong answers provided by the 
student per question* Since a student Is not precisely wrong or right 
In CAAPM, these data are not collected* A capability needs to be de- 
veloped In CAAPM to collect and analyze relevant data — for example, 
scores — on students. 



Appendix A 
THE EXTERNAL VALIDITY GRAPH* 



Even though the student may learn to express his own probabilities 
accurately, the question remains as to how those probabilities relate 
to reality. That question can be answered In part by his external 
validity graph [8]. 

The external validity graph can be used to determine how accu- 
rately the student assesses the predictive validity of his facts and 
reasons. It can be thought of as his "track record" as„ an assessor. 
We may collect all those events that he claims have a probability of, 
say, 0.80 of occurring and determine what percentage of them did In 
fact occur. If he were a perfect assessor, we would expect roughly 
80 percent of them to occur. Figure A.l Illustrates such a graph when 
the number of observations Is unlimited. The diagonal AB represents 
a perfect assessor. If we collect, say for 20 questions, all the prob- 
abilities used by a student and compute the relative frequency with 
which each particular probability is associateid with a correct alter- 
native, we may construct an external validity graph for that student 
taking that test. We then assume that the relationship between stu- 
dent response and relative frequency can be approximated satisfactorily 
by a straight-line simple linear regression. 

We call the resulting line the student's realism line. The lines 
A'B' and A"B" in Fig. A.l illustrate possible results. The line A'B' 
has the following interpretation: About 60 percent of the events that 
were given a probability of 1.0 of occurring did in fact occur. We 
interpret this to mean that the student tends to overvalue his facts 
and reasons. Similarly, interpretation of the line A"B" shows that 
the events that were given a probability of 0.60 (or greater) of oc- 
curring did in fact occur 100 percent of the time. In other words, 
the student tends to undervalue his facts and reasons. We may further 
Interpret the realism line in the following way: "When he responded 

This appendix is taken from Re£. 1, pp. 9-12. 
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Assigned probability 



Fig. A.I — External validity graph and realism lines. 

X he should have responded with Ax + B," where Ax + B Is the equation 
of his realism line. We may then transform the student's probabilities, 
using his realism line, and recalculate his score on the basis of the 
new probability assignments. This revised score can then b6 v^ffed to 
decompose his original score Into the portion attributable to bias In 
his assignment of probabilities and the portion attributable to lack 
of information about the subject matter itself. The lower section of 
Fig. A. 2 Illustrates such a score decomposition. 

The scores are actually reported as the difference between the 
student's scores and the maximum attainable for a test with a specified 
length. The overall gain is that maximum less the original score. The 
gain attainable from more information is that maxinmm less the revised 
score. The gain from more "realistic" use of probabilities is simply 
the revised score less the original score. 



Complete mastery 




No mostery 



You tend to overvalue your information. 

You can improve your score by 37 points 
by more realistic use of your knowledge. 

You can improve your score by 224 points 
by more study. 

You can improve your score by 261 points 
overall. 

Fig. A. 2^ — Score interpretation display 



Appendix B 
SAMPLE TESTS 



rand4: A Test on International Relations 

rand6: A Test on Statistics 

rand7: A Test on Art 

randS: A Test on Music 

rand9: A Test on Literature 



Lesson: rand4 



A TEST ON INTERNATIONAL RELATIONS 
correct answer 

Selling goods for export at prices lower than those charged domes- 
tic buyers is an example of: 

*a. dumping 

b. subsidy 

c. embargo 

The communist theorists expound today's world movements as a 
triumph for the revolutionary concept of: 

a. Marx 
*b • Lenin 

c. Stalin 

Identify the most important policy advisory body that works di- 
rectly with the Secretary of Defense: 

*a. Armed Forces Policy Council 

b. National Security Council 

c. Defense Research Engineering Group 

The decision as to which countries the U.S. shall recognize di- 
plomatically lies with the: 

a. Senate 

b. Secretary of State 
*c. President 

What do we call the strategy of buying goods to prevent a rival 
nation from getting them? 

a. loading 
*b. preemptive buying 
c* embargo 



k n 
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Lesson: randS 
A TEST ON STATISTICS 

* correct answer 

1. The least square estimator for the model Y-XB-KJ Is best since It: 

a. Is unbiased 

b. Is consistent 

*c. has smallest variance 

2. In matrix algebra, the transpose of a sum Is Just the: 

*a. sum of the transposes 

b. product of the transposes 

c. sum of the products 

3. An F (R.S) distribution (I.e., an F with R and S degrees of free- 
dom Is the ratio of) : 

a. chl-squared with R degrees of freedom and a normal with S 
degrees of freedom 
*b. two chl-squareds with R and S degrees of freedom 
c. two T-dlstrlbutlons with R and S degrees of freedom 

4. A matrix Is singular If It: 

a. has no linearly dependent rows or columns 
*b. has at least two linearly dependent columns 
c. has a nonzero determinant 

5. In testing whether or not there are increasing returns to scale in 
an estimated Cobb-Douglas production function, you would want to 
use a: 

a. two tailed F test 
*b. one tailed T test 
c. one tailed chl-squared test 

6. In the model Y-XB-HJ under conditions of heteroscedastldty , the 
OLS estimator of B Is not: 

a. unbiased 

b . consistent 
*c. efficient 
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In the model Y"XB-HJ with M observations and G explanatory vari- 
ables plus a constant term* S-squared (I.e., sum of squared re- 
siduals divided by degrees of freedom) Is distributed as a: 

*a. chl-squatiL:d with M-»6-l degrees of freedom 

b. T-dlstrlbution with N-G-1 degrees of freedom 

c. chl-squared with N-G degrees of freedom 

In the case of heteroscedastlclty, you would want to estimate the 
coefficients by: 

a. ordinary least sqiiares 
*b. weighted least squares 

c. two-stage least squares 

It Is possible to obtain a large F value (I.e., all the coeffi- 
cients are significantly different from zero) and a low R-squared 
if you have: 

*a, many observations and few parameters 

b. few observations and few parameters 

c. few observations and many parameters 
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Lesson: rand? 
A TEST ON ART 

correct answer 



1. 



Of the following categories, the one in which visual perspective 
was most thoroughly explored is: 

*a. late Renaissance paintings 
b« Byzantine mosaics 
c. gothic stained glass 

2. Of the following, the watercolorist noted for seascapes is: 

A.^ Theodore Gericault 
*b, Winslow Homer 
c. Jacob Epstein 

3. Which of the following Dutch artists is noted for his genre paint- 
ings? 

*a. Jan Steen 

b, Jan Vermeer 

c. Jacob van Ruisdael 

4. Of the following painters, the one who is best known for paintings 
of children is; 

*a. Mary Cassatt 

b. Rosa Bonhettr 

c. Georges Braque 

5. The drawings of William Blake were based on: 

*a. imaginative scenes 

b. contemporary scenes 

c. urban life 

6. A technique in which no colors are mixed but tones are built up 
by a stipple of pure brilliant color is called: 

*a. pointillistic 

b. expressionistic 

c. impressionistic 

7. "The Night Watch" was painted by: 

a. Franz Hals 

b. Titian 

*c. Rembrandt van Rijn 
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ART 



2 

The French artist who used the entertainment world almost exclu- 
sively as subject matter for his paintings was: 

*a. Henri de Toulouse-Lautrec 

b. Francois Boucher 

c. Georges Seurat 

Foremost among the so-called action painters was: 

a. Salvador Dali 

b. Paul Klee 

*c, Jackson Pollock 
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Lesson: randS 
A TEST ON MUSIC 

* correct answers 

1. An unaccompanied solo passage In a concerto, usually brilliant and 
virtuoso In style, is called a: 

*a. cadenza 
b. coda 
c • counterpoint 

2. The "Emperor Concerto" for piano and orchestra was composed by: 

a. Frederic Chopin 

b. Franz Liszt 

*c. Ludwig van Beethoven 

3. The romance of Rodolfo and Mlml is a prominent story thread in the 
opera: 

*a. "La Boheme" 

b. "La Traviata" 

c. "II Trovatore" 

4. The "book" detailing the sequence of events in an opera is called 
the: 

a. synopsis 

b. score 
*c. libretto 

5. Among the following, the instrument that does not use a reed is 
the: 

a. oboe 
*b. flute 

c . bassoon 

5. The composer who wrote a famous "Ave Maria," using a prelude from 
Bach's "Well-Tempered Clavier" as an accompaniment, was: 

a. Franz Schubert 
*b. Charles Gounod 
c. Gabriel Faure 

^ Which composer wrote a symphony depicting the emotional stress of 
a city under siege? 

a. Sergei Rachmaninoff 
*b . Dmitri Shos takovich 
c. Jean Sibelius 
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MUSIC 
2 

Vloletta is a character in the opera: 

*a. "La Traviata" 

b. "II Trovatore" 

c. "Rigoletto" 

According to current expert opinion, which one of the following 
is regarded as an important jazz composer and arranger? 

a. Count Basie 
*b. Duke Ellington 
c. Benny Goodman 



ERIC 



46 



-40- 



Lesson: rand9 



A TEST ON LITERATURE 



1. 



3. 



4. 



5. 



6. 



- correct answers 

Which of the following poets reached his highest attainment In 
satire: 

*a. Alexander Pope 

b. William Blake 

c. John Milton 

Nlccolo Machlavelll, In his political treatise "The Prince," 
advocates, as the basic policy of efficient government: 

a. aggressive expansion 

b. rigid economy 

*c. diplomatic expediency 

The poem "Lapis Lazuli" was written by: 

a. T. S. Eliot 

b. Dylan Thomas 

*c. William Butler Yeats 

"My head Is bloody, but unbowed" epitomizes the mood of the poem 

*a. "Invlctus" 

b. "Lay of the Last Minstrel" 

c. "Tfir^ldler" 



A literary Work that tells about a hero who slays a dragon Is: 

*a. "Beowulf" . 

b. "The Battle of Brunanburh" 

c. "The Wanderer" 

Macbeth 's tragic flaw can best be described as: 

*a. lust for power 

b. jealousy 

c. overbearing pride 

"A Valediction Forbidding Mourning" Is a poem by which of the 
following metaphysical poets? 

*a. John Donne 
b* Andrew Marvell 
c. George Herbert 
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LITERATURE 



2 

The name of Upton Sinclair's novel "Dragon Harvest" calls to mind 
an incident in the story of: 

a. Hercules 

b . Perseus 
*c. Jason 

The greatest of the Greek comic dramatists, noted for his satires 
of the life of his time, was: 

a . Menander 
*b . Aristophanes 

c. Euripides 
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